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DETAILED ACTION 
Response to Arguments 

1 . The amendment filed 5/1 6/2007 have been entered and made of record. 

2. The Applicant canceled claims 7 and 9. 

3. In response to the amendments filed on 5/16/2007: 

The "Objections to the specification" have been entered and therefore the 
Examiner withdraws the objections to the specification. 

The "Objections to the claims" have been entered and therefore the Examiner 
withdraws the objections to the claims. 

The "Claim rejections under 35 U.S.C. 112, second paragraph" have been 
entered and therefore the Examiner withdraws the rejections under 35 U.S.C. 1 12, 
second paragraph. 

4. Applicant's arguments with respect to claims 1-6, 8, and 10-1 1 have been 
considered but are moot in view of the new ground(s) of rejection. 

5. Applicant's arguments filed 5/16/2007 have been fully considered but they are 
not persuasive. 

The Applicant alleges, "In contrast to claim 1 ..." in page 7 and "In addition, the 
Simpson reference is not understood ..." in page 7, and states respectively that the 
Simpson art reference teaches away from classifying images as claimed in claim 1 and 
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teaches classifying pixels instead. However the Examiner disagrees because Simpson 
clearly states that the classification is for the "individual images" and for the "sequences 
of images" [see abstract, lines 8-10, page 2138, section 1) Single Image Classification, 
page 2139, section 2) Image Sequence Classification]. Although Simpson discusses 
pixel classification, this pixel classification is intended to support the overall method of 
creating an overall classification of the entire image itself. 

The Applicant alleges, "In addition, the Simpson reference is not understood ..." 
in page 7, and states respectively that Simpson classifies pixels instead of images and 
that Simpson doesn't teach or suggest performing an initial image classification for each 
image in a sequence and that Simpson doesn't teach or suggest performing a revised 
image classification for each image in the seguence as required by claim 1 . However 
the Examiner disagrees initially because Simpson does disclose clearly that the 
classification is for images as discussed above. The Examiner also disagrees secondly 
because Simpson does teach classification for each image in the sequence and 
creating a revised image classification for each image in the sequence. Simpson 
discloses providing a sequence of about twelve daylight scene images per day and 
generating an independent classification [Feed Forward Neural Network (FFNN) or in 
other words the single FFNN or in other words the three layer Feed Forward Neural 
Network (NNCCS): no feedback, see page 2139, Fig. 3a, abstract, lines 8-10] for each 
of the individual single images provided from this twelve daylight scene images 
sequence. Simpson also discloses providing this twelve daylight scene images per day 
sequence and generating a "time series" dependent sequence classification [Recurrent 
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Neural Network (RNNCCS): feedback, see page 2139, Fig. 3b, section 2, paragraph 1, 
"time series" shows that the temporal succession of the sequence is considered when 
inputting data into the Recurrent Neural Network, abstract, lines 8-10]. Simpson also 
clearly states that the Recurrent Neural Network (RNNCCS) is "more accurate than a 
single FFNN" [see page 2139, section 2, paragraph 2 "THE RNNCCS network lines 
6-8] which shows a comparison of the classification of the twelve daylight scene images 
sequence between the two networks is considered and that these networks indeed do 
classify each image in this twelve scene per day sequence. Therefore, the Single 
FFNN is used as an initial classification for each of this twelve daylight scene images 
individually [no feedback] and the Recurrent Neural Network is used as a more accurate 
or a revised type of classification of each of the twelve daylight scene images sequence 
[with this time series dependent input using feedback]. Therefore the rejections using 
the Simpson art reference will be maintained. 

In response to the amended limitation "at least two pairs of consecutive images 
in the sequence of images having different elapsed times between their capture" of 
claim 1, the Examiner believes that Simpson suggests this limitation when stating that 
the "temporal sampling (hourly or faster ) provides about twelve daylight scenes per 
day". This statement of sampling hourly or faster gives reason that the sampling could 
be done evenly (time elapsed hourly) or unevenly (time elapsed differently, hourly or 
faster meaning an elapsed hour between one consecutive pair and an elapsed half-hour 
for the next consecutive pair etc.). But to clear any uncertainty which may arise, the 
Examiner has incorporated a second reference Loui et al (US 6,606,411 B1) to 
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specifically disclose that the time difference between consecutive or adjacent images 
are unevenly spaced (different elapsed times between consecutive image pairs). 

Claim Rejections - 35 USC § 103 

6. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 1 02 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

7. The text of those sections of Title 35, U.S. Code not included in this action can 
be found in a prior Office action. 

8. Claims 1-2 and 4-5 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Simpson ("A recurrent neural network classifier for improved retrievals of areal 
extent of snow cover" - IEEE - vol 39, Oct 2001 , pages 21 35-2147, applied in prior 
Office Action) in view of Loui et al (US 6,606,41 1 B1 ). 

Re Claim 1: Simpson discloses a method / single image classification using feed- 
forward neural networks (FFNN) and image sequence classification using recurrent 
neural networks (RNNCCS) (see pages 2138-2139, section B. Present Approaches, 
abstract, lines 8-10) for improving scene classification of a sequence of digital images / 
sequence of twelve daylight scenes per day (see page 21 39, section 2, paragraph 1 ) 
comprising the steps of (a) providing a sequence of images / sequence of twelve 
daylight scenes per day captured in temporal succession / temporal sampling into time 
series (see page 2139, section 2 - Image Sequence Classification Using Recurrent 
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Neural Networks, first paragraph); (b) classifying each of the images individually / Single 
Image Classification using Feed-Forward Neural Networks (FFNN) based on 
information / spectral and textural contained in the individual image / single image using 
no feedback to generate an initial image classification / single image classification for 
each of the images / sequence of twelve daylight scenes per day (Feed Forward Neural 
Network (FFNN) or in other words the single FFNN or in other words the three layer 
Feed Forward Neural Network (NNCCS): each image is individually classified [no 
feedback], see page 2139, Fig. 3a, abstract, lines 8-10); and (c) imposing a pre- 
determined temporal context model / Recurrent Neural Network (RNNCCS) on the 
sequence of images / sequence of twelve daylight scenes per day temporal sampling 
into time series images to generate a revised image classification / more accurate than 
a single FFNN for each image / sequence of twelve daylight scenes per day, wherein 
the pre-determined temporal context model considers the temporal succession / time 
series of the sequence of image / sequence of twelve daylight scenes per day 
(Recurrent Neural Network (RNNCCS): time series dependent sequence classification 
[feedback], see page 2139, Fig. 3b, section 2, paragraph 1, "time series" shows that the 
temporal succession of the sequence is considered when inputting data into the 
Recurrent Neural Network, abstract, lines 8-10, page 2139, section 2, paragraph 2 "THE 
RNNCCS network lines 6-8, "more accurate than a single FFNN" shows a more 
accurate or revised classification occurs and also that a comparison of the classification 
of the twelve daylight scene images sequence between the two networks [Single FFNN 
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and Recurrent Neural Network] is considered and that these networks indeed do 
classify each image in this twelve scene per day sequence). 

However, Simpson fails to specifically suggest that at least two pairs of 
consecutive images in the sequence of images having different elapsed times between 
their capture. 

Loui discloses at least two pairs of consecutive images / adjacent pictures in the 
sequence of images having different elapsed times / uneven time interval or time 
difference between their capture (see Figs. 2 and 4, col. 3, lines 38-41 , col. 4, lines 27- 
29, the time interval or the time difference between adjacent pictures are uneven as 
seen in the Figures where some adjacent pictures have 1 minute in elapsed time 
between them while other adjacent pictures have 2, 4, 6, etc. minutes in elapsed time 
between them). 

Therefore, it would be obvious to one of ordinary skill in the art at the time the 
invention was made to modify Simpson's method using Loui's teachings by including 
the ability to have consecutive images with different elapsed times in order better relate 
adjacent pictures with clustering (see Loui, col. 3, lines 36-41 ). 

Re Claim 2: Simpson further discloses the information used in step (b) includes pixel 
information / spectral and textural (see page 2138, section 1 - Single Image 
Classification Using Feed-Forward Neural Networks, first paragraph). 
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Re Claim 4: Loui further discloses the pre-determined temporal context model in step 
(c) is independent of elapsed time between consecutive images (see Figs. 2 and 4, col. 
3, lines 38-41, col. 4, lines 27-29, col. 1, lines 59-60; there could be an instance when 
no time interval data is available). 

Re Claim 5: Simpson further discloses the pre-determined temporal context model in 
step (c) is dependent on elapsed time / temporal sampling into time series between 
consecutive images (see page 2139, section 2 - Image Sequence Classification Using 
Recurrent Neural Networks, first paragraph, the Recurrent Neural Network uses the 
twelve daylight scenes per day evenly hourly spaced sequence of images in a time 
series manner for further improvement in classification). 

9. Claim 3 is rejected under 35 U.S.C. 103(a) as being unpatentable over Simpson 
as modified by Loui, and in further view of Tretter et al (US 6,977,679 B2, applied in 
prior Office Action). The teachings of Simpson as modified by Loui have been 
discussed above. 

However, Simpson as modified by Loui fails to specifically suggest the 
information used in step (b) includes capture-device generated metadata information. 

Tretter, as recited in claim 3 , discloses the information used in step (b) includes 
capture device generated / digital camera metadata / focusing distance information (see 
abstract, lines 6-13). 
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Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to further modify Simpson's method, as modified by Loui, 
using Tretter's teachings by including focusing distance metadata to Simpson's step (b) 
in order to enhance the classification of snow cover by further distinguishing clouds and 
snow cover by identifying the difference in distance between clouds and the snow cover 
from the satellite which captures images. 

10. Claims 6, 8, and 10-1 1 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Simpson as modified by Loui, and further in view of Huang 
("Integration of multimodal features for video scene classification based on HMM" - 
IEEE - Sept 1999, pages 53-58, applied in prior Office Action). The teachings of 
Simpson as modified by Loui have been discussed above. 

However, Simpson as modified by Loui fails to teach that the temporal context 
model is a causal Hidden Markov Model, that the temporal context model is a non- 
causal model, that the temporal context model is imposed using a Viterbi algorithm, and 
that the temporal context model is imposed using a belief propagation algorithm. 

Huang, as recited in claim 8 . discloses the pre-determined temporal context 
model is a non-casual model / discrete ergodic Hidden Markov Model dependent on 
both the previous image and a subsequent image / visited from any state (see page 55, 
section Product HMM, page 56, section SIMULATION RESULTS, second paragraph, 
the discrete ergodic HMM visits any states or images from any state or image which 
makes the model non-causal). 
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Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to further modify Simpson's method, as modified by Loui, 
using Huang's teachings by replacing a non-causal discrete ergodic Hidden Markov 
Model with Simpson's temporal context model in order to give more correlation data 
between all the images of the sequence instead of just a few images (images before the 
current image) to further improve the accuracy of the classification. 

Although Simpson's temporal context model, as modified by Loui, and in further 
view of Huang's modifications teaches a non-causal discrete ergodic Hidden Markov 
Model, it does not specifically disclose, as recited in claim 6 . that the temporal context 
model is the causal Hidden Markov Model dependent on a previous image, and it does 
not specifically disclose, as recited in claim 10 . that the temporal context model is 
imposed using Viterbi algorithm, and it does not specifically disclose, as recited in claim 
11, that the temporal context model is imposed using a belief propagation algorithm. ■ It 
would have been obvious to one of ordinary skill in the art at the time the invention was 
made though to have such a feature of causality in a HMM model, a Viterbi algorithm, or 
a belief propagation algorithm for a temporal context model because they are just other 
methods of computing the probability for classification of a particular sequence which 
Huang's non-causal Hidden Markov Model is basically accomplishing. 
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Conclusion 

1 1 . The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. Sun discloses MyPhotos which is a system for home photo 
management and processing. 

12. Applicant's amendment necessitated the new ground(s) of rejection presented in 
this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP 

§ 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 
CFR 1136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the date of this final action. 

1 3. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Bernard Krasnic whose telephone number is (571) 270- 
1357. The examiner can normally be reached on Mon-Thur 8:00am-4:00pm and every 
other Friday 8: 00am-3: 00pm. 
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If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Jingge Wu can be reached on (571) 272-7429. The fax phone number for 
the organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-91 99 (IN USA OR CANADA) or 571-272-1 000. 

Bernard Krasnic 
July 6, 2007 




